That is the question that must be at the core of any truly scientific endeavor – is the apparent or alleged phenomenon we are dealing with actually real? That may seem obvious, and with some scientific disciplines it is a given. This is not necessarily the case, however, with medical science. Medicine strives to be an applied science, so it asks a lot of proxy questions like “Does it work?”
This may seem like the same question, and I would argue that it should be, but it is extremely easy to get distracted by how we measure and study clinical outcomes so as to miss this point. This is because we first have to operationally define “work”. When I say a treatment works, what does that mean? Unfortunately, to many, it may mean that someone feels subjectively better when they take the treatment, but that doesn’t cut it.
One way to define Science-Based Medicine is by how we define what it means for a science-based treatment to work. It means that when we look critically at the totality of scientific evidence, it is more likely than not (to some reasonable level of confidence) that there is a specific measurable physiological effect from the treatment. We spend a lot of time fleshing out the details of how to do such an analysis. Here is a quick summary:
First, the treatment has to be at least reasonably plausible (a factor that distinguishes SBM from evidence-based medicine – EBM). That does not mean we have to know the precise mechanism of action but it should mean the treatment does not break fundamental laws of physics, or defy a basic understanding of chemistry or physiology, or necessarily introduces a purely magical phenomenon.
We also need high-quality clinical evidence for efficacy. This means that there are experimental trials which isolate the alleged phenomenon as completely as possible from all other confounding factors. Generally in medicine we refer to these confounding factors as placebo effects – an apparent response to the non-specific aspects of being treated, and therefore such trials are called placebo-controlled.
Before we consider a medical phenomenon to be real I would argue that we need such placebo-controlled trials that are sufficiently rigorous in their design and execution that show a statistically significant and clinically meaningful objective outcome (as objective as possible) that sufficiently replicates. Replication must also be independent – not just confined to one lab, set of researchers, or country. Further, the effect persists over time (does not vanish due to the decline effect), and persists when we look at only the highest quality studies. The effect also persists when controlled for publication bias. I would also now include the stipulation that the effect persists when you only look at studies that have been pre-registered, in order to minimize the effects of P-hacking.
Is that too high a bar? Absolutely not – but with some caveats. In medicine it is extremely important to know that a clinical phenomenon is really real, because not only does that form the basis of treating people, it also helps to confirm or refute our understanding of the underlying biology. That understanding is then used to develop and refine further treatments. If we think a phenomenon is real when it isn’t, that can launch an entire branch of research and clinical practice that is disconnected from reality. That represents a profound waste of limited resources, in addition to potential massive harm caused to patients.
The caveat is that the threshold for using a medical intervention is not the same as the threshold for concluding that scientifically a conclusion is solid and we can now use it as a premise for further research and development. Clinical decision making is based mostly on risk vs benefit, and that analysis often takes place with incomplete scientific information. It is reasonable, for example, to use an experimental treatment on a patient with an otherwise terminal illness.
While a risk vs benefit analysis may favor the conclusion that a particular treatment is reasonable to use (for now), that should not be conflated with the conclusion that some specific mechanism of action is a real phenomenon.
Having said that, I also think that the totality of the evidence strongly suggests that the medical profession as a whole needs to recalibrate its assessment of when a treatment is reasonable to use. The primary problem is that there is a disconnect between the actual predictive value of preliminary evidence, and the perceived predictive value. Specifically, apparently positive preliminary evidence that a treatment works has a much lower predictive value than most clinicians believe. This results in a high percentage of “reversals”, where a common practice based on preliminary evidence is stopped because later more definitive evidence is negative.
These are all fixable problems. We can reconsider where to set the traditional P-value before considering a study positive, to shift some of the false positives toward the true negative. We can universally require pre-registering clinical trials. Journals can require statistical analysis other than p-values, focusing more on effect sizes, number needed to treat, and Bayesian analysis. They can also publish more negative studies, and more exact replications. They can require greater rigor, such as internal replications prior to publishing (again, trying to weed out some of the false positives). Preliminary studies can come with explicit warnings about their preliminary nature and their lack of predictive value. Also the standards of EBM systematic reviews should be higher, and align more with the approach of SBM.
Finally I will point out that all of this is not just theoretical. We see in the real world of medicine what happens when the profession fails to adequately ask and answer the question – is the phenomenon actually real? The proliferation of acupuncture is a great example. I won’t go into the details again here, I have written about it extensively in SBM and elsewhere. Suffice to say, the evidence strongly suggests that acupuncture is not real – there is no underlying phenomenon specific to acupuncture. In fact, acupuncture points do not even exist. Yet it has infiltrated almost every medical discipline, and stands as a prime example of what happens without adequate SBM standards. There are plenty of other examples, such as EMDR in mental health practice.
We must also remember that the world of medicine does not only contain honest brokers. There are those who seek to exploit the weaknesses in the system to promote snake oil and take advantage of desperate patients and a gullible public.